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ABSTRACT 

One of the most significant challenges facing systems of 
collective intelligence is how to encourage participation 
on the scale required to produce high quality data. This 
paper details ongoing work with Phrase Detectives^ an 
online game-with-a-purpose deployed on Facebook, and 
investigates user motivations for participation in social 
network gaming where the wisdom of crowds produces 
useful data. 



INTRODUCTION 

Recent advances in human language technology have 
been made possible by acquiring large-scale resources 
created by novel methods of harnessing collective intel- 
ligence over the Internet. This approach is motivated 
by the observation that a group of individuals can con- 
tribute to a collective solution, which has a better per- 
formance and is more robust than an individual's so- 
lution. This is demonstrated in simulations of collec- 
tive behaviour in self-organizing systems (Johnson, Ras- 
mussen, Joslyn, Rocha, Smith & Kantor 1998). 

Collective Intelligence (CI) systems such as Wikipedia 
and similar large initiatives have shown that a surprising 
number of individuals can be willing to participate in 
projects. 

A novel approach was the development of games- wit h-a- 
purpose (GWAP) that aggregate data from non-expert 
players for collective decisions similar to what might be 
expected from an expert. 

Encouraging participation in GWAP projects remains a 
significant challenge, despite successful early efforts (von 
Ahn 2006). 

This paper investigates user participation in CI systems 
and whether social network platforms could offer any- 
thing to this approach. The paper then goes on to 
describe Phrase Detectives^ a GWAP for creating an- 
notated language resources, specifically looking at the 
interface that was developed for the Facebook platform 
and the modifications that were made to maximise the 
social incentives for players. Data from over a year of the 
game being live on Facebook is analysed and, in conclu- 
sion, suggestions are proposed for developers considering 
using a social networking platform for deployment of CI 
interfaces. 



COLLECTIVE INTELLIGENCE SYSTEMS 

Collective intelligence can be shown in many domains in- 
cluding Computer Science, Economics and Biolog}Qbut 
here we focus on coordinating collective action in compu- 
tational systems of CI. Individual decisions made by the 
community are aggregated in an attempt to produce a 
high quality, collective decision comparable to an expert 
judgement. 

One important goal of CI systems is to overcome the bot- 
tleneck in creating and maintaining resources that would 
normally have to be done by paid administrators. Exam- 
ples include encyclopedia websites like Wikipedia, citizen 
science projects for common knowledge and games-with- 
a-purpose that collect metadata. 

By collecting decisions from a large, distributed group 
of contributors it is possible to approximate a sin- 
gle expert's judgements (Snow, O'Connor, Jurafsky & 
Ng 2008). This is in line with findings in other contexts 
(Feng, Besana & Zajac 2009, Albakour, Kruschwitz & 
Lucas 2010). 



Participation 

The willingness of Web users to collaborate in the cre- 
ation of resources is clearly illustrated by Wikipedia. En- 
glish Wikipedia numbers (as of October 2011) 3,773,941 
articles, written by over 15.5 million collaborators and 
5,559 reviewersj^ 

Citizen science projects, where non-expert volunteers 
complete large-scale or computationally difficult tasks, 
include Open Mind CommonsensJ^ (now ConceptNet|^ 
which demonstrated that Web collaboration can be re- 
lied on to create resources, with 14,500 volunteers con- 
tributing nearly 700,000 sentences (Singh 2002). 

The first, and perhaps most successful, game-with-a- 
purpose was The ESP Gam^ which attracted over 
200,000 players who have produced over 50 million labels 
(von Ahn 2006). 



^http: / / scripts. mit.edu/^cci/HCI 

^http:/ /meta. wikimedia.org/wiki/List_of_Wikipedias 

^http:/ /openmind. media. mit.edu 

^http: / / conceptnet.media.mit.edu 

^http:/ /www. gwap.com/gwap 
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Welcome to Phrase Detectives 

This is no ordinary Facebook game! 

Lovers of language and literature; this is the place 
where you can work together to improve future 
generations of language technology. By indicating 
relationships between words and phrases you will help 
to create a resource that is rich in linguistic information, 

So don't worry if you spend hours playing this gamOj 
you'll be doing it in the name of Science, 




01 Nov 10 

Here are a few game stats for you number junkies, We 
have over 1,5 million examples of human language in the 
database submitted by 3500 players^ a collaborative 
effort of over 2700 hours or 112 days. On average 
players are quicker to disagree with other players than 



WIN PRIZES IN OCTOBER 

Congratulations to all our previous winners. We're 
continuing the monthly high scoring prize during 
October 20 11 where you could win £50 for being 
the top scorer of the months as well as other 
prises for being in the top 10, 

To enter simply play the game and start scoring 
points. Winners will be contacted by email so if y 
want to win please add your email on your Settings 
page. Full details are in the FAQ, 

Good luck! 



Highest level player 
Julie Jones (Ivl 13) 

Highest rated player 
100<V;> Shona MclNeill (Ivl 7) 

96% " j| Jean McNeill (Ivl 12) 

967o HjemmaAnneBastable(lvlll) 

96% H Eileen McNeill (Ivl 7) 
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Finland has Northern 
Lights liks no other 
place. Watch the video 
and like us! 



Like ■ 26^008 people like this. 
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You too can find gold in 
Crans-Montana! 
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Free Sky+HD box with 
free std set-up, up to 
20Mb broadband, 
inclusive calls* and a 
£25 M66 voucher when 
you join with Sky-HHD 
online. 



1* Chat (Offline) 



Figure 1. Screenshot of the Phrase Detectives Facebook homepage. 



Clearly there is huge potential for the general public to 
become engaged in CI systems and collaborate in pro- 
ducing resources that would not be possible to achieve 
using traditional methods. 



Social Networking Platforms 

Given the social nature of CI it seems logical to deploy 
CI systems on platforms where the users are already net- 
worked. In recent years social networking has become 
the dominant pastime online. As much as 22% of time 
online is spent on social networks like Facebook, MyS- 
pace, Bebo, Twitter and others. This is three times the 
amount of time spent emailing and 7 times the amount 
of time spent searching the Internet 

The success of social network games such as Cityville, 
with over 50 million active players each month, or The 
Sims, Farmville and Texas HoldEm Poker, with over 30 
million active monthly players each, show that the poten- 
tial for large scale participation is possible using social 
networking platforms Q 

An estimated 927 million hours are spent each month by 



^http: / / mashable.com/2010 /08/02/ stats-time-spent-online 
^http:/ /www. appdata.com 



Facebook users playing game^ which is another indica- 
tor of the vast human resource available. 

A study of US and UK social network users showed that 
Facebook was by far the most frequently used platform 
for social network gaming (used by 83% of users, com- 
pared to MySpace, the next highest platform, at 24%) 
However Google are planning to increase their market 
share of this valuable resource 

Human language technology games integrated into so- 
cial networking sites such as Sentiment Quif^ on Face- 
book show that social interaction within a game environ- 
ment does motivate players to participate (Rafelsberger 
& Scharl 2009). 

It is becoming more apparent that CI interfaces should 
be linked to social networking sites like Facebook to 
achieve high visibility, to explore different ways play- 
ers can collaborate and to exploit this enormous human 
resource. 



^ http : / / www . allfacebook. com / facebook-games-st at ist ics- 
2010-09 

^ http : / / www . infosolut ionsgroup . com / 20 1 0_PopCap_Social 
_Gaming_Research_Results.pdf 

""^^ http : / / www.reuters.com / article /20 11/08/12/ us-google- 
games-idUSTRE77A66H20110812 
http: / / apps.facebook.com/sentiment-quiz 
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Rhinogradentia (Wikipedia) 

Rhinogradentia (also known as snouters or Rhinogrades or Nasobames) is a fictitious mammal 
order documented by the equally fictitious German naturalist Harald Stumpke. The order's 
most remarkable characteristic was the Nasorium^ an organ derived fi'om the ancestral 
species's nose, which had variously evolved to fijifill every conceivable function. 

Both the animals and the scientist were allegedly creations of Gerolf Steiner, a zoology 
profiessor at the University of Karlsruhe. A mock taxidermy of a certain Snouter can be seen at 
the l^usee zoologique in Strasbourg. 

The order's remarkable variety was the natural outcome of evolution acting over millions of 
years in the isolated Hi-yi-yi islands in the Pacific Ocean. 



NAME THE CULPRIT 

Has the pinrase shown in orange been mentioned before in this text or is it a property of 
another phrase? Select the closest phrase(s) within the text if it has been mentioned before 
and click "Done". 



Not mentioned before 



n This is a property «iS 
Done $ 



-jre i 



C5 Comment on this phrase 
C^5kip this one 

C^Skip - closest phrase can't be selected 
i^5kjp - closest phrase is no longer visible 
^ Skip - error in the text 



Figure 2. Detail of a task presented in Annotation Mode. 



PHRASE DETECTIVES FACEBOOK INTERFACE 

The first interface for tiie Phrase Detectives gamj^ 
(Ctiamberlain, Poesio & Krusciiwitz 2008) enabled groups 
of players to work on the same task over a period of 
time as this was likely to lead to a collectively intelligent 
decision (Surowiecki 2005). 

The Facebook version of Phrase Detective^^ launched 
in February 2011, maintained the overall game architec- 
ture whilst incorporating a number of new features de- 
veloped specifically for the social network platform (see 
Figure [T]) . 

The game was developed in PHP SDK (a Facebook API 
language allowing access to user data, friend lists, wall 
posting etc) and integrates seamlessly within the Face- 
book site. Data generated from this version of the game 
is compatable with previous versions and both current 
implementations of the game run simultaneously on the 
same corpus of documents. 

In order to play the game a Facebook user must grant 
certain permissions: the basic access (user details and 
friends list), which is required for all applications, and 
access to posting on the user's wall. Once the user has 
allowed the game access they never need to login to the 
game, only to Facebook. 

The game uses 2 styles of text annotation for players to 
complete a linguistic task. Initially text is presented in 
Annotation Mode (called Name the Culprit in the game - 
see Figure [2]). This is a straightforward annotation mode 
where the^ayer makes an annotation decision about a 



Rhinogradentia (Wil<ipedia) 

Rhinogradentia (also known as snouters or Rhinogrades or Nasobames) is a fictitious mammal 
order documented by the equally fictitious German naturalist Harald Stumpke. The order's 
most remarkable characteristic was the Nasorium^ an organ derived fi'om the ancestral 
species's nose, which had variously evolved to fijifill every conceivable fijnction, 

Both the animals and the scientist were allegedly creations of Gerolf Steiner, a zoology 
professor at the University of Karlsruhe, A mock taxidermy of a certain Snouter can be seen at 
the Musee zoologique in Strasbourg, 

[The order's remarkable variety was the natural outcome of evolution acting over millions of 
years in the isolated Hi-yi-yi islands in the Pacific Ocean. 



DETECTIVE CONFERENCE 

Another detective has said the phrase in orange has been mentioned before and its 
nearest mention is highlighted in blue. Do you agree with them? 



Disagree 



Agree 




^^http: / / www.phrasedetectives.com 

^^http: / / apps.facebook.com/phrasedetectives 



Figure 3. Detail of a task presented in Validation Mode. 

highlighted markable (section of text). If different play- 
ers enter different interpretations for a markable then 
each interpretation is presented to more players in Vali- 
dation Mode (called Detectives Conference in the game 
- see Figure [3|. The players in Validation Mode have to 
agree or disagree with the interpretation. 

Players are trained with training texts created from a 
gold standard (a text that has been annotated by a lin- 
guistic annotation expert). A player always receives a 
training text when they first start the game and may also 
need to complete one when being promoted to the next 
level. Once the player has completed all of the training 
tasks they are given a rating (the percentage of correct 
decisions out of the total number of training tasks). The 
rating is recorded with every future annotation that the 
player makes as the rating is likely to change over time. 

The scoring system is designed to reward effort and mo- 
tivate high quality decisions by awarding points for ret- 
rospective collaboration (see Figure |4|. 

The game makes full use of socially motivating factors 
inherent in the Facebook platform. Any of the player's 
friends who are playing the game form the player's team, 
which is visible in the left hand menu. Whenever a 
player's decision agrees with a team member they score 
additional points. 

Player levels have well-defined criteria and the player 
must activate the new level once the criteria are met 
(see Figure |5|: 

• Total points scored 

• The player's rating 

• Documents completed 

• Training documents completed 

• Facebook posts made from the game 
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Will Rogers phenomenon (Wikipedia) 

Case work points 2 
During this case you agreed with: 
1^ 5arah Ridgeway 1 

■ Gary H Amos 1 # 

H Jemma Anne Bastable 1 ^ 

B Livio Robaldo 2 

-.'f Jean McNeill 3';# 
Julie Jones 1 (tf* 

Other detectives 6 #^ 

Post to my wall €^ 
Get your newt case U 



Figure 4. Detail of the reward screen, displayed at the 
end of each case, showing the player how many points 
they scored and who they agreed with. 



The Facebook game has monthly and ah-time leader- 
boards as well as leaderboards for the highest level play- 
ers, highest rated players and the players with the biggest 
team. 

INCENTIVES 

There are 3 main incentive structures that can be 
used to encourage recruitment and participation: per- 
sonal; social; and financial (Chamberlain, Poesio & 
Kruschwitz 2009). All incentives are applied with cau- 
tion as rewards have been known to decrease annotation 
quality (Mrozinski, Whittaker & Furui 2008). 

A previous survey showed that women are more likely to 
play social network games than men^ and this could have 
an impact on the types of incentives offered. Facebook 
allows access to user data, including gender, as part of 
the basic access for the application so gender and work- 
load of the players could be investigated 

Personal incentives 

Personal incentives are evident when simply participat- 
ing is enough of a reward for the user. Generally, the 
most important personal incentive is that the user feels 
they are contributing to a worthwhile project. 

Also important for the players of Phrase Detectives is 
that they read texts that they find interesting. The 
choice of documents is important in getting users to par- 
ticipate in the game, to understand the tasks and to keep 
playing. Whilst some texts are straightforward, others 
can provide a serious challenge of reading comprehension 
and completion of linguistic tasks. 

Texts were graded on complexity (on a scale of 1 to 4) 
on import. A player can choose the maximum level of 
document complexity they wish to read as they may be 

^^It was assumed for the purposes of this investigation that 
Facebook users declare their gender truthfully. 




Jon Chamberlain 

Bookworm (level 5) 

You have: 

# 6649 
^ 293 

V 96% 

5 



Figure 5. Detail showing criteria for the next level, dis- 
played to the player on their homepage. 



motivated to play the game to improve their English 
skills or, equally, because they enjoy reading challenging 
texts. 

Social incentives 

Social incentives reward users by improving their stand- 
ing amongst their peers (in this case their fellow players 
and friends). 

Using leaderboards and assigning levels for points can 
be an effective motivator, with players often using these 
as targets i.e., they keep playing to reach a level or high 
score before stopping (von Ahn & Dabbish 2008). 

To investigate this in Phrase Detectives^ players were 
grouped by how much progress they had made towards 
the next level in terms of points they had scored beyond 
the requirement for their current level. For example if 
a player had 110 points on level 2 (where the points 
requirement is 100) and the level 3 requirement is 200 
points, then this player has progressed 10% towards the 
next level. 

News feed (or wall) posting is integrated into the game. 
This allows a player to make an automatically generated 
post to their news feed which will be seen by all of the 
player's friends (see Figure [g]) 

The posts include a link back to the game. Players are 
required to make a post from the game every time they 
are promoted to the next level. Posting is a very im- 
portant factor in recruiting more players as studies have 
shown that the majority of social game players start to 
play because of a friend recommendation.^ 

Posts may be social (display information about the doc- 
ument the player is working on or has just completed), 
collaborative (asking friends to join the game) or com- 
petitive (position in a leaderboard) . Social posts are 
similar to information social network users share with 
friends so it is reasonable to assume they will be the 
most common type of post made from the game. This 

Since the release of the game Facebook has changed how 
posts are displayed. Posts from the game now appear on the 
player's profile and in a news ticker. 

"^^http:/ /www. lightspeedresearch. com/press-releases/it 's- 
game-on-for-facebook-users 
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Update status Add Photo / Video ^ Ask question 



What's on your mind? 



NEW TOP STORV 



4- 1 MORE RECENT STORV 



n 



5am O'Neill 

is working on Deep-fried Mars Bar by WiWpedia in the Phrase Detectives game. 
Uke • Comment ' Play the game ■ about a minute ago via Phrase Detectives ■ 



Figure 6. Detail of a news (or wall) post created auto- 
matically from the game, as seen by the player's friend. 




19 28 37 46 55 64 73 
Players ranked by total points scored 
—Total points per player 



was investigated by analysing the logs of wall posts made 
from the game. 

Financial incentives 

Financial incentives reward effort with money. When 
financial rewards were offered in the game they were for 
monthly high scores, ranging from £50 for first place, 
£15 for second place, £10 for third place and £5 for 
fourth to tenth place. During July 2011 additional daily 
lottery-style prizes of £5 were awarded, along with £20 
for the player with the highest level, highest rating and 
largest team. The monthly prizes motivate the high- 
scoring players to compete with each other by doing more 
work and motivate low-scoring players in the early parts 
of the month when the high score is low. The prizes were 
sent as Amazon vouchers by email to the winners. 

Whilst financial incentives are important to recruit new 
players, a combination of all 3 types of incentives is essen- 
tial for the long term success of a game (Smadja 2009). 

The effectiveness of incentives was analysed by looking 
at new players, active players and new annotations each 
month. Months where prize funds were available were 
compared to months where there was none and a per- 
annotation cost effectiveness was calculated. For the first 
5 months no prize funds were offered but the following 
months all had prize funds of £110, except July 2011 
which had £320 (see Table [l]). 

The site was promoted in February, July and December 
2011 to email lists, existing players, relevant Facebook 
groups etc, as well as advertised on Google Ads and Face- 
book (a modest total pay-per-click budget of £160) and 
competition websites. 

RESULTS 

Results from the Facebook interface of Phrase Detectives 
were analysed from February 2011 to February 2012. 

Gender and workload of players 

The Facebook game attracted 612 players of which 63% 
were female, 35% were male and 2% did not disclose their 
gender. Of the ten highest scoring players, 60% were 
female, 30% were male and 10% did not disclose their 
gender. This supports the previously mentioned survey 
that social network games are played predominately by 
women. 



Figure 7. Chart showing the scores of players. 




1 10 19 28 37 46 55 64 73 82 91 
Players ranked by total points scored 

Total annotations perplayer 



Figure 8. Chart showing the total annotations of players, 
ranked by their total score. 

In a study of the previous version of Phrase Detectives it 
was reported that the ten highest scoring players (repre- 
senting 1.3% of total players) had 60% of the total points 
on the system and had made 73% of the annotationj^ 
(Chamberlain et al. 2009). In the Facebook version of 
the game the ten highest scoring players (representing 
1.6% of total players) had 89% of the total points and 
had made 89% of the annotations (see Figure [t]). The 
same ranking was used in Figure [S] to show that although 
the total number of annotations of players is generally 
in line with points scored it is not always the case. 

These results show that the majority of the workload 
(in this case annotation of documents) is being done by 
a handful of players. However, the influence of players 
who only contribute a little should not be undervalued 
as in some systems it can be as high as 30% of the work- 
load (Kanefsky, Barlow & Gulick 2001) and this is what 
makes the collective decision making robust. 

A subset of 112 players (those that declared their gender, 
had a rating above zero and had completed at least one 
annotation) were analysed to investigate whether gen- 
der was related to the amount of work a player does. 
Of these players men represented 35% of the total (39 
players). On average male players had completed 1,290 
annotations and scored 4,636 points, compared to female 



^ For the purpose of data analysis, annotations and valida- 
tions are counted together and referred to as annotations. 
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Figure 9. Chart showing the distribution of players' 
progress towards the next level. 



players who had completed 4,636 annotations and scored 
20,776 points. Although female players appear to be do- 
ing more work on average the difference is not statisti- 
cally significant (using an unpaired t-test for significance 
testing). 

The ten highest scoring male players were then compared 
to the ten highest scoring female players. On average, 
the former had completed 4,817 annotations and scored 
17,628 points, whereas the latter had made 48,359 anno- 
tations and scored 144,905 points. The difference in both 
workload and score is statistically significant (p<0.05). 

This suggests that not only are female players more likely 
to play a socially networked game, they are also more 
likely to actively participate than male players. Further 
analysis is required to investigate whether female players 
provide higher quality annotations. 

Using levels as goals 

All players who had a score (i.e. they had passed the 
training stage and had completed a document) were cat- 
egorised in terms of their progress to the next level. In 
the first analysis all players from levels 1 to 17 were in- 
cluded. 

To distinguish between players who gave up after com- 
pleting the training (i.e., the early stage of level 1) from 
the players stopping after reaching their goal of the next 
level, a second analysis was made excluding players on 
level 1. 

The clustering of players near to the early stages of 
progress to the next level (see Figure [9| could be an 
indication that players are motivated to work towards 
the next level before stopping, however the level cri- 
teria and scoring system make this difficult to assess. 
Players score points in batches because all points scored 
on a document are added when the case is completed. 
Also, at this time, all extra points from other players 
who have agreed with their annotations are added. This 
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■ Aboutthe current document 

■ When the documentis completed 

■ When the player is promoted 

■ To directly invite friends 

■ When the player rating is updated 

■ Monthly leaderboard position 
All time leaderboard position 



Figure 10. Chart showing the breakdown of 423 wall 
posts made by players. 



effect becomes more negligible at higher levels where the 
progression to the next level is longer. 

The analysis looks at player progression at the end of the 
data collection period i.e., players who get to the next 
level then never play the game again. To investigate this 
in detail the progression of each player session should 
be plotted however the game is not designed to test this 
and, if it were, many more players would be needed. 

Whilst it is intuitive to think that players will use level 
thresholds as goals this evidence does not support it. 

Posting from the game 



Players' posts (see Figure 10) were most commonly social 
i.e., about the document the player was working on or 
had just completed (52%). This compares to competitive 
posts when they went up a level (13%), when their rating 
was updated (10%) or to post about their position in the 
leaderboard (12%). The remaining 13% of news posts 
were players making a direct collaborative request for 
their friends to join the game. 

These results support the assumption that players are 
most likely to make posts from the game when the infor- 
mation is similar to what they might usually post. This 
should be a design consideration when developing a CI 
system for social networks. 

The effect of incentives on recruitment and participation 
Figure pT] shows the growth of Phrase Detectives on Face- 
book. Months where there was active promotion of the 
site (February, July and December 2011) show increases 
in new players, as one would expect. The month with the 
largest prize fund also had the most recruitment, double 
that of a later month with a smaller prize fund. 

Based on the assumption that the first promotion month, 
when the site went live, was an exception as players of 
the previous game joined the new version, there is an 
indication that financial incentives increase recuitment 
to the game if sufficiently advertised. 

It is noticable that the amount of active players (a player 
who made more than one annotation in a particular 
month) stayed consistent and does not seem to increase 
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Figure 11. Chart showing new annotations plotted against new players and active players. Prizes were available in the 
game from July 2011 to February 2012 (see Table [TJ. * indicates a month with active promotion for the game. 



with recruitment or financial incentives. Whilst it could 
be expected that the number of active players steadily 
increases over time as more players are recruited, the 
results show that most players will play the game for 
a short period of time and only small number continue 
to play every month. This is confirmed by the previous 
results that only a handful of players do most of the 
work. 

Financial incentives do appear to be a strong motivat- 
ing factor when considering how much work the active 
players do. Months with prizes have considerably more 
new annotations than those without, but with a similar 
number of active players. 

This suggests that active players are motivated by finan- 
cial incentives, however the large amount of gameplay in 
October and November 2011 indict ae that other moti- 
vating factors such as personal and social incentives are, 
to some extent, also successful. 



Cost effectiveness of financial incentives 
The cost per annotation is a measure of the effectiveness 
of the prize fund. A baseline of 4699 new annotations per 
month could be expected without a prize fund (the aver- 
age of the first 5 months' new annotations) so anything 
over that could be attributed to the prize fund. 

The average cost per annotation across the months where 
there was a prize fund was £0.0028 (see Table [T]) and this 



can be compared to other CI systems. Amazon Mechan- 
ical Turk (MTurkJ^ is often used as a way of getting 
data quickly but can cost £0.007 - 0.70 ($0.01 - 0.10) 
per annotation (Mason & Watts 2010). 

Both systems require the annotations to be aggregated 
before a collectively intelligent answer can be produced 
however even professional annotation schemes require 
some degree of validation. 

The advantage of GWAP over MTurk is that personal 
and social incentives can be used, as well as financial, to 
minimise the cost and maximise the persistence of the 
system. 

It is also worth considering the setup and maintenance 
costs of CI systems in a cost per annotation analysis. 

CONCLUSIONS 

In addition to accessing the vast human resources that 
social networking platforms offer, CI systems can also 
take advantage of the inherent social structure and 
shared personal data available to maximise the incen- 
tives that encourage participation. 

Phrase Detectives has shown some valuable insights into 
user motivations in social network gaming and participa- 
tion in CI efforts. The results support previous surveys 
that show women are more likely to play, and will spend 
more time playing, socially networked games. 

^^http:/ /www. mturk.com 
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Table 1. Cost per annotation based on the prize fund 



Month 


Prize fund 


Cost / annotation 


Feb - Jun 2011 


- 


- 


Jul 2011 


320 


0.0022 


Aug 2011 


110 


0.0058 


Sep 2011 


110 


0.0042 


Oct 2011 


110 


0.0008 


Nov 2011 


110 


0.0012 


Dec 2011 


110 


0.0025 


Jan 2012 


110 


0.0028 


Feb 2012 


110 


0.0031 



There are indications that players want to share infor- 
mation from the game that is similar to the information 
usually shared in their social networks. The success of a 
socially networked game relies on creating an experience 
that players want to share with their friends. 

The results suggest that attracting and motivating the 
right kind of player is as important as attracting lots of 
players because, although collective intelligence needs a 
crowd, that crowd also needs to do some work. Financial 
incentives, coupled with promotion, increase recruitment 
and have a considerable impact on participation. 

The increase in data collected from the game due to fi- 
nancial incentives still makes it a cost effective alterna- 
tive to other CI systems such as MTurk. 
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